Papers with weakly-supervised scenarios
MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on phrase localization uses caption-image datasets as weak supervision . existing work on supervised phrase localisation uses a large-scale annotated dataset . |
| Approach: | They develop a multimodal alignment framework to leverage more widely available caption-image datasets to model phrase relevance. |
| Outcome: | The proposed model improves on the widely-adopted Flickr30k dataset . it also improves the previous best unsupervised result by 5.56% . |
Pre-training Cross-Modal Retrieval by Expansive Lexicon-Patch Alignment (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent large-scale vision-language pre-training relies on image-text global alignment by contrastive learning and is further boosted by fine-grained alignment in a weakly contrastive manner for cross-modal retrieval. |
| Approach: | They propose expansive lexicon-patch alignment (ELA) to align image patches with a vocabulary rather than only the words explicitly in the text for annotation-free alignment and information augmentation. |
| Outcome: | The proposed method outperforms state-of-the-art methods on cross-modal retrieval and can learn representative fine-grained information. |